Talking Pictures: Indexing and Representing Video with Collateral Texts

The relevance of collateral texts for building knowledge-based visual information systems is discussed, with reference to moving images of dance. The knowledge acquisition technique Protocol Analysis is applied to elicit verbal reports from experts watching moving images with complex contents and interwoven meanings. These reports are analysed at lexical, clausal and discourse levels, using text analysis methods. The results show a potential for using these reports to index and represent moving images, through the creation of lexical resources and knowledge-bases. The KAB system integrates text analysis modules with a video object database to process collateral texts.